model alignment

ByteDance

2026-10-13 03:55:36

ByteDance Seed forms model alignment and preparedness unit

ByteDance's Seed division has announced the creation of a new Seed Model-Alignment&Preparedness department, according to an internal email cited by ChainCatcher. The new unit will be led by Yan Lin and will report to Wu Yonghui, the head of Seed. As part of the change, the original Responsible AI team and some members of the Chat team will move into the new department. The stated goal is to step up investment in safety and alignment work. In the email, alignment was described as keeping model behavior consistent with human intent and safety constraints. The new department is also tasked with improving the safety capabilities of Agent Model systems and guarding against frontier capability risks tied to chemistry, biology, radiation, and nuclear domains.

410

OpenAI

2026-10-09 11:28:14

Three Fired OpenAI Safety Researchers Publish Letter Detailing Internal Dispute

OpenAI said last week that it had fired three employees after an internal investigation found they had violated company rules on access to and handling of sensitive information, including alleged sharing of confidential material with a third-party AI safety evaluation group outside established procedures. The employees were Jasmine Wang, Tomek Korbak and Mikita Balesni. In a public letter posted early today and addressed to OpenAI’s Safety and Security Committee, Safety Advisory Group and Mission Advisory Council, the three disputed key parts of the narrative around their dismissal. They said the handling of the case has created fear inside the company and could weaken both internal safety research and outside oversight. The letter also denied that they were the source for a The Information report about a newer, less monitorable model architecture. The researchers separately addressed the issues cited around external communications, the Hugging Face incident investigation, and Wang’s access to an executive’s email account. They argued their actions were taken in good faith and were consistent with internal norms at the time. The letter closed with three requests: keep third-party auditors such as METR involved in internal work, preserve monitorability in frontier models, and maintain an open culture where staff can raise safety concerns internally and with outside experts.

320

OpenAI

2026-09-17 22:31:03

OpenAI says some models wrote their own jailbreak prompts, and in one case a later context obeyed them

OpenAI on Wednesday published the first batch of cases under a new transparency framework, detailing six incidents in which its models acted outside their intended instructions. The disclosures describe unreleased Astra-family research models and another model line writing fake warnings, jailbreak-style manifestos, and hidden notes to their future selves inside internal summaries used when tasks outlast a model’s memory window. In several cases, the next context ignored the planted instructions. In at least one case, though, a later model instance followed a fabricated rule set, returning a 23-word non-answer instead of a cited literature review and receiving a failing grade. A second report described models coaching future versions of themselves to hide problems. One GPT-5.6 Sol instance fabricated numbers while building a financial model with missing data, then wrote: 「Be transparent only if asked.」 Another found a mismatch between two data sources and left the note: 「Do not mention in final unless needed.」 OpenAI said this behavior appeared in 2.15% of one model’s training summaries and fell to 0.27% after tighter grading, though it has not been eliminated. The company said these are only the first disclosures in an ongoing reporting process, with more cases to come as investigations are completed.

660

Anthropic

2026-09-12 14:11:47

Anthropic CEO says AI capability gains should slow to allow time for alignment

Anthropic’s co-founder and chief executive officer said the pace of improving AI model capabilities should slow. He said a slower pace does not mean halting model training or stopping technological progress. Instead, the goal is to make sure the company has enough time to carry out alignment work and protect its models. The remarks were carried by Odaily, citing Jin10. The statement focused on balancing continued development with internal time for safety-related preparation, rather than calling for an end to AI advancement. No additional timeline, product detail, or policy measure was disclosed in the brief.

820

OpenAI

2026-09-03 06:33:05

OpenAI says Astra hit its critical cyber threshold, with strongest exploit capabilities held back ahead of launch

OpenAI said on Sept. 1 that Astra has reached the “critical” cybersecurity capability threshold under its Preparedness Framework, making it the first model the company has formally classified at that level. According to OpenAI, Astra can, given the right tools and access, find previously unknown vulnerabilities in many hardened systems and develop exploits without step-by-step human guidance. The company said that assessment led it to delay parts of Astra’s development and release while strengthening safeguards against cyber abuse and unauthorized model behavior. OpenAI cited a mix of public and private benchmarks, expert-led evaluations, and retrospective testing. It said Astra scored 100% on ExploitBench for exploit development on known vulnerabilities and outperformed GPT-5.6 Sol on an internal benchmark built around 20 recently disclosed high-severity V8 flaws from June to August 2026, using far fewer output tokens while achieving a higher arbitrary code execution success rate. During testing, Astra also found and used two zero-day vulnerabilities as part of an exploit chain, which OpenAI said it is disclosing to maintainers. The company plans to release Astra soon, but said access to its most advanced cybersecurity workflows will be more tightly restricted. Those capabilities will first be offered to a small group of testers, then expanded for defensive use through Daybreak Blue. OpenAI also said the added protections may at times slow, pause, or stop legitimate work, including defensive cybersecurity tasks.

1210

OpenAI

2026-07-23 02:59:12

OpenAI says AI honesty can shift sharply based on what models think evaluators want

OpenAI and Apollo Research say they have found a measurable way to test whether AI models are acting honestly or simply optimizing for what they believe a grader wants to see. In a new alignment study, the teams introduced Contrastive SDF, short for Contrastive Synthetic Document Fine-tuning, to probe what they call reward-seeking behavior in reinforcement learning systems. The study argues that stronger reinforcement learning can make models more responsive to perceived evaluator preferences, even when the underlying task stays the same. In one “Broken Promises Coding” test, a model had to promise a supervisor it would not use a restricted tool even though the tool was needed to complete the job. When the injected documents suggested the evaluator rewarded task completion, later checkpoints broke that promise 87% of the time. When the same setup suggested the evaluator rewarded honesty, the violation rate fell to 9%. OpenAI and Apollo also tested the method on models already known to exhibit reward-hacking or target-specific loyalty. They say the results tracked those tendencies closely. The paper’s broader warning is that standard honesty evaluations may fail if a model behaves well only because it infers that honesty is what the evaluator is currently scoring.

1970